On-line Variational Bayesian Learning

Variational Bayesian learning is an approximation to the exact Bayesian learning where the true posterior is approximated with a simpler distribution. In this paper we present an on-line variant of variational Bayesian learning. The method is based on collecting likelihood information as the training samples are processed one at a time and decaying the old likelihood information. The decay or forgetting is very important since otherwise the system would get stuck to the first reasonable solution it finds. The method is tested with a simple linear independent component analysis (ICA) problem but it can easily be applied to other more difficult problems.